Zipf and Heaps Laws' Coefficients Depend on Language

We observed that the coefficients of two important empirical statistical laws of language – Zipf law and Heaps law – are different for different languages, as we illustrate on English and Russian examples. This may have both theoretical and practical implications. On the one hand, the reasons for this may shed light on the nature of language. On the other hand, these two laws are important in, say, full-text database design allowing predicting the index size. Introduction. Perhaps the most famous statistical distribution in linguistics is Zipf law [1, 2]: in any large enough text, the frequency ranks (starting from the highest) of wordforms or lemmas are inversely proportional to the corresponding frequencies: log fr ≈ C – z log r (1) where fi is the frequency of the unit (wordform or lemma) having the rank r, z is the exponent coefficient (near to 1), and C is a constant. In a logarithmic scale, it is a straight line with about – 45° angle. Another, less famous but probably not less important empirical statistical law of language is the Heaps law: the number of different wordforms or lemmas in a text is roughly proportional to an exponent of its size: log ni ≈ D + h log i (2) where ni is the number of different units (wordforms or lemmas) occurring before the running word number i, h is the exponent coefficient (between 0 and 1), and D is a constant. In a logarithmic scale, it is a straight line with about 45° angle. The nature of these laws is not clear. They seem to be specific for natural languages in contrast to other types of signals [3]. In practice, knowing the coefficients of these laws is important in, for example, full-text database design, since it allows predicting some properties of the index as a function of the size of the database. In this paper, we present the data that show that the coefficients of both laws – z and h – depend on language. For our experiments, we use English and Russian texts. Experiments with Spanish (which we do not discuss here) gave the results between those for English and Russian. * The work was done under partial support of CONACyT, REDII, and SNI, Mexico. We thank Prof. R. Baeza-Yates, Prof. E. Atwell, and Prof. I. Bolshakov for useful discussion. 1 We ignore Mandelbrot’s improvements to Zipf law [1] since they do not affect our discussion. Experimental data. We processed 39 literature texts for each language, see Appendix 2, chosen randomly from different genres, with the requirement that the size be greater than 10,000 running words (100 KB); total of 2.5 million running words (24.8 MB) for English and 2.0 million (20.2 MB) for Russian. We experimented with wordforms and lemmas, with very similar results. We plotted on the screen the graphs for pairs of texts (one English and one Russian), using for Zipf law the points: xr = log r, yi = log fr (xi = log i, yi = log ni for Heaps law). The difference in the angle was in most cases clearly visible. We used linear regression to approximate such a graph by a straight line y = ax + b, where a and b correspond to – z and C for Zipf law, or h and D for Heaps law. Since the density of the points (xi, yi) increases exponentially with xi, we scaled the distance penalty for regression by i x c − (we have to omit here the details; obviously, the results do not depend on c), which gave the following formulae for a and b: